In order to make intra prediction more precise and reduce redundancy more effectively, 35 intra prediction modes are used in an HEVC (High Efficiency Video Coding) standard. When intra coding is performed, however, and if Rate-Distortion optimization (RDO) is successively conducted on the 35 intra prediction modes to determine an optimal prediction mode, an encoder would be unable to manage the computation for conducting RDO in all prediction directions, and thus the determination of the optimal prediction mode cannot be achieved. For this purpose, a “Rough-to-Fine” mode selection scheme is used in the HEVC. But, in such a scheme, the selection for the optimal intra prediction mode would still constitute the majority of workload in the entire coding process. Therefore, a method which can both ensure coding quality and improve the efficiency of intra prediction mode selection is desired.